Conceptualising and evaluating teacher quality: Substantive and methodological issues

Whereas findings from recent research highlight the importance of teacher quality in improving students’ academic performances and experiences of schooling, substantive and methodological issues surrounding the conceptualisation and evaluation of teacher quality are not wellunderstood. Such deficiencies are particularly evident in claims for ‘findings’ derived from econometric research – especially from those studies that merely employ conceptualisations and proxy ‘measures’ of quality in terms of teachers’ qualifications, experience, and students’ academic outcomes. Moreover, the econometric models fitted to the available, mostly aggregated data, typically fail to conceptualise and ‘measure’ teacher quality in terms of what teachers should know (subject-matter knowledge) and be able to do (pedagogical skill). Nor do such models account for the measurement, distributional and structural properties of the data for response and explanatory variables – failings that all too frequently yield misleading interpretations of findings for both policy and practice. Following brief introductory comments related to current contexts, the paper focuses on two approaches towards the resolution current deficiencies – both of which have important implications for conceptualising and evaluating teacher quality, namely: (a) capacity building in teacher professionalism grounded in evidence-based pre-service teacher education content and subsequent in-service professional development, and (b) the specification and evaluation of teaching standards. The paper concludes by arguing that since the most valuable resources available to any school are its teachers, there is a crucial need for both a substantive and methodological refocus of the prevailing economic teacher-quality/student-performance/merit-pay research and policy agenda to one that focuses on the need for capacity building in teacher professionalism (and its evaluation) in terms of teaching standards related to what teachers should know and be able to do. Introductory comments Consistent with the adoption of corporate management models in educational governance and the prevailing climate of outcomes-driven economic rationalism in which such models operate, policy activity related to issues of: accountability, assessment, standards monitoring and benchmarking, performance indicators, quality assurance, teacher quality, school and teacher effectiveness, are widespread. However, political, economic and industrial issues surrounding educational effectiveness are sensitive, despite the level of non-partisan political consensus (at least in Australia) regarding the macro and micro economic importance of teacher quality and quality teaching for equipping students adequately to meet the constantly changing demands the modern workplace (e.g., Bishop, 2007; Macklin, 2006; Nelson, 2002, 2004). The global economic, technological and social changes under way, requiring responses from an increasingly skilled workforce, make high quality educational provision an imperative – 1 Correspondence related to this paper should be directed to: Dr Lawrence Ingvarson, Principal Research Fellow, ACER, Private Bag 55, Camberwell, VIC 3124 (Email: [email protected]); OR to Dr Ken Rowe, Research Director, Learning Processes research program, ACER, Private Bag 55, Camberwell, VIC 3124 (Email: [email protected]). 2 For example, see: Access Economics (2005); Alton-Lee (2002, 2005); Curtis and Keeves (2000); Fenstermacher and Richardson (2005); Hanushek (1971, 1986, 2004); Ingvarson and Kleinhenz (2006a,b); Kleinhenz and Ingvarson (2004); Marsh, Rowe and Martin (2002); OECD (2005, 2006); Rowe (2001, 2004a, 2005a,b, 2006a,b); Rowe and Stephanou (2003); Rowe, Stephanou and Hoad (2007). _____________________________________________________________________________________ The Economics of Teacher Quality conference, ANU: 5 February 2007 Conceptualising & Evaluating 2 Ingvarson & Rowe Teacher Quality _____________________________________________________________________________________ especially high quality teaching. Although OECD education ministers have committed their countries to the goal of raising the quality of learning for all, this ambitious goal will not be achieved unless all learners, irrespective of their characteristics, backgrounds and locations, receive high-quality teaching (OECD, 2001, 2005). Since teachers are the most valuable resource available to both schools and higher education institutions in the realisation of this goal, an investment in teacher quality and on-going professionalism is vital. In our view, this goal can only be realised by ensuring that teachers are equipped with subject-matter knowledge and an evidenceand standards-based repertoire of pedagogical skills that are demonstrably effective in meeting the developmental and learning needs of all students for whom they have responsibility – regardless of students’ backgrounds and intake characteristics, and whether or not they experience learning difficulties. Despite the emphasis placed on the importance of teacher quality and quality teaching in recent OECD publications, as well as similar emphases underlying the 2001 No Child Left Behind Act in the USA (see: Center on Education Policy, 2003; LaTrice-Hill, 2002; US Department of Education, 2002), the bulk of international scholarly discourse concerned with educational effectiveness has largely ignored the importance of specifying evidence-based standards for instructional effectiveness and their evaluation for teacher registration, accreditation, and on-going professional development (Rowe, 2007a). With few exceptions, especially from the related school effectiveness research literature (e.g., Mortimore, 1991; Reynolds, Creemers et al., 2002), discussions that focus on the constituent elements of teacher quality in terms of what teachers should know and be able to do (i.e., instructional effectiveness, or the what and how of quality teaching), are conspicuous by their absence. Rather, the dominant emphasis continues to be characterized by offerings advocating structural changes for systemic reform, including curriculum reconstruction, single-sex schooling, class size (see Hattie, 2005b) etc., that have a long and not-so-distinguished history of rarely penetrating the classroom door. A note about methodological limitations endemic to econometric research focussing on the link between teacher quality and student academic performance is appropriate here (e.g., Hanushek, 1971, 2004; Leigh & Ryan, 2006; Monk, 1992; Podgursky, Monroe & Watson, 2004; Rivkin, Hanushek & Kain, 2005). Since these limitations are well established, they need little reiteration here. In brief, however, an extensive body of work indicates that the typical single-level econometric models fitted to the available data employing general linear model (GLM) techniques under ordinary-least-squares estimation procedures, are inappropriate on at least two counts. First, they fail conceptualise, measure and evaluate teacher quality in terms of what teachers know and do. Second, such models rarely account for the measurement, distributional and structural properties of the data for response and explanatory variables – oversights that all too frequently yield misleading interpretations of findings for both policy and practice. Failures to account for the inherent hierarchical structure of the data are especially problematic. Findings from fitting explanatory multilevel models to relevant data (at the 3 See: Coltheart and Prior (2007); Darling-Hammond and Bransford (2005); Farkota (2003, 2005); Hattie (1987, 2003, 2005a); Hoad, Munro et al. (2005, 2007); Purdie and Ellis (2005); Rowe (2005a,b, 2006a, 2007a); Slavin (2005); Stronge (2002); Westwood (2006); Wheldall (2006). 4 For examples of exceptions, see: Bond, Smith et al. (2000); Bosker, Kremers and Lugthart (1990); Darling-Hammond and Baratz-Snowden (2005); Darling-Hammond and Bransford (2005); Fenstermacher and Richardson (2005); Fullan, Hill and Crévola (2006); Ingvarson (2001); Ingvarson and Kleinhenz (2006a,b); Rowe (2002; in press a,b). 5 For relevant examples, see: Embretson and Hershberger (1999); Goldstein (1997, 2003); Goldstein and Spiegelhalter (1996); Hill and Rowe (1996, 1998); Masters (2004b); Masters and Keeves (1999); Millmann (1997); Raudenbush and Bryk (1988); Raudenbush and Willms (1991, 1995); Rowe (2000, 2004b, 2006b, 2007b); Rowe and Hill (1998). _____________________________________________________________________________________ The Economics of Teacher Quality conference, ANU: 5 February 2007 Conceptualising & Evaluating 3 Ingvarson & Rowe Teacher Quality _____________________________________________________________________________________ student, class/teacher, and school levels) consistently indicate that in excess of 40 percent of the residual variance in measures of student performance (adjusted for students’ background and intake characteristics) is at the class/teacher-level (see citations given in footnote 5; and for key findings from meta-analytic syntheses of more than 500,000 evidence-based studies, see Hattie, 2003, 2005a). These findings are especially useful. By identifying that the major sources of residual variation in students’ learning and achievement progress are at the class/teacher level, they assist in specifying and evaluating teacher quality in terms of what quality teachers know and are able to do. Moreover, such findings constitute invaluable data for informed, evidencebased content of pre-service teacher education and subsequent in-service professional development (Ingvarson, 1998, 2000, 2003: Rowe KS, Pollard & Rowe KJ, 2005), as well as for the specification and evaluation of teaching standards (Ingvarson & Kleinhenz et al., 2006a-c). Rather than focussing on the economics of teacher quality, per se (as presented by other contributors to this conference), the present paper stresses the need for policies and processes designed to improve teacher quality through building teacher capacity, including the need for valid methods of specifying and evaluating teacher quality, as well as teaching standards. While such policies and processes have universal applicability, this paper focuses on the urgent need for the adoption of these policies and procedures throughout Australian education systems. The need for valid methods of assessing teacher quality Pronouncements on the importance of teacher quality to student learning outcomes usually recognise the need to place greater value on teaching if the profession is to attract and retain high quality graduates from schools and universities (DEST, 2003; Ramsey, 2000). The major argument of this paper is that we will find it difficult to place greater value on teaching in substantive ways, such as better salaries and career paths for accomplished teachers, unless we greatly improve the capacity of the profession to define, evaluate and certify high quality teaching. For a detailed review of national and international approaches to evaluating and rewarding accomplished teaching, see Ingvarson and Kleinhenz (2006a). Policies with respect to teacher quality fall into two main groups – policies designed to affect the composition of the teacher workforce, and policies designed to improve the capacity of individual teachers. Strategies in both areas are obviously important. Australia shares the problem of attracting and retaining a necessary share of the best graduates from schools and universities (OECD, 2001, 2005a). A recent synthesis of research on attitudes to teaching as a career found that extrinsic factors such as remuneration, workload, employment conditions and status were the most significant factors influencing able graduates not to choose teaching, and to leave the profession (DEST, 2006). If the ability of the teaching profession to compete with other occupations for the best graduates is to increase, research findings indicate that teaching salaries relative to those in related professions is the most importance factor (e.g., Dolton, Chevalier & McIntosh, 2001), especially relative salaries after ten to fifteen years in the job. This paper focuses mainly on policy strategies related to improving teacher quality through building capacity (rather than composition), though it is recognised that these two strategies overlap. Strategies designed to improve career paths and rewards for good teaching, for example, may aim to affect both composition and capacity if rewards are linked to evidence of knowledge and skill via professional development. Whereas indicators of composition typically focus on administrative and demographic data such as SES, TERs and GPAs, indicators of capacity focus on what teachers know and do in schools and classrooms. Why do we need better methods for measuring teacher quality? The 2006 edition of the OECD’s report, Education at a Glance (OECD, 2006), indicates that whereas the average ratio of the salary at the top of the incremental scale is 1.70, it is only 1.47 in Australia, and nearly 3 in Korea and Japan. The typical salary scale for teachers in Australia does not place high value on evidence of teacher quality. Consequently, it is a weak instrument for improving student achievement. It does not provide incentives for professional development nor reward evidence _____________________________________________________________________________________ The Economics of Teacher Quality conference, ANU: 5 February 2007 Conceptualising & Evaluating 4 Ingvarson & Rowe Teacher Quality _____________________________________________________________________________________ of attaining high standards of performance. This ratio seems unlikely to improve unless further salary increments are linked to evidence of enhanced teacher knowledge and skill. Thirteen of 32 OECD countries report that they adjust the base salary of teachers on the basis of outstanding performance in teaching, or successful completion of professional development activities. Australia is not one of them. While progression to the top of the salary ladder is rapid in Australia – it takes only 9 years for most Australian teachers to reach the top of the scale compared with 24 years on average in OECD countries – there are no further career stages based on evidence of attaining higher levels of teaching standards. The implicit message in most Australian salary scales is that teachers are not expected to improve their performance after nine years. We suggest that the profession needs clearer guidelines as to what it expects its members to get better at with experience. Indeed, the salary scale provides few incentives for continued development of expertise in teaching. Indeed, for teaching, the relationship between evidence of professional development and salary progression is weak. A survey of public opinion about teacher quality in the USA found that all groups recognised the importance of teacher quality and strongly support reforms that lead to significant increases in teacher salaries, if those reforms also provide better guarantees that these increases reward evidence of professional development and quality teaching (Hart & Teeter, 2002). Public attitudes in Australia are probably similar. Guarantees of quality teaching, however, will be meaningless without valid methods of measuring teacher performance. Nonetheless, there has been renewed discussion about performance-based pay in Australia as a means of placing greater value on teaching. A review of research in this area by the Australian Council for Educational Research (ACER), commissioned by the Australian Government Department of Education, Science and Training (DEST), indicates that the reason for so many failed merit pay schemes over the past thirty years has been the lack of understanding about the complexity of developing valid and professionally credible methods for gathering data about teaching and assessing teacher performance (Ingvarson, Kleinhenz & Wilkinson, forthcoming). Unlike most other professions, the teaching profession has found it difficult to create a strong market for highly accomplished practitioners. A major reason for this is that the profession has yet to develop a voluntary system for providing certification to teachers who attain high standards of performance, at least one that employing authorities find credible and useful (Ingvarson & Kleinhenz, 2006a,b). There are many highly accomplished teachers, but no profession-wide system by which they can gain a highly respected and portable certification of their accomplishments. Consequently, incentives for teachers to provide evidence of skills via professional development through stages of increasing expertise are weak. Despite the paucity of incentives, there are strong indications that many in the profession wish to move down this path. A stronger market for highly accomplished teachers may be critical in areas of teacher shortage. This is partly why the Australian Science Teachers Association and the Australian Association of Mathematics Teachers have developed their own standards for highly accomplished teachers in recent years (Brinkworth, 2006; Semple & Ingvarson, 2006). Several other subject associations are undertaking similar initiatives. School systems within Australia are also looking for better ways to recognise and retain good teachers, such as Western Australia with its Level 3 Classroom Teacher scheme. The ACER review on performance-based pay (noted above) found evidence that there is a stronger demand – in the sense of a greater capacity to offer over-award payments – for highly accomplished teachers in independent schools. The NSW Association of Independent Schools is introducing a system of remuneration based on increasing levels of professional standards (Newcombe, 2007). This applies at the entry level as well. This year (2007), all graduates of the highly selective Graduate Diploma of Education for secondary teachers from the University of Western Australia (UWA) accepted positions in non-government schools. _____________________________________________________________________________________ The Economics of Teacher Quality conference, ANU: 5 February 2007 Conceptualising & Evaluating 5 Ingvarson & Rowe Teacher Quality _____________________________________________________________________________________ Other major related challenges are to ensure greater equity in the distribution of highly accomplished teachers across schools and school systems. At present we know that out-of-field teaching is more likely to be found in rural, remote and disadvantaged schools, but we do not know how equitable the distribution of quality teachers is across schools. Without valid measures of teacher quality, we cannot conduct research on the contribution that variation in teacher quality might make to Australia’s comparatively high levels of variation in student learning outcomes in schools for students drawn from high to low socioeconomic status backgrounds, as revealed in recent international studies of student achievement such as Australia’s participation in the OECD Programme for International Student Achievement (PISA), and in the IEA Trends in International Mathematics and Science Study (TIMSS). Effective teacher education is essential to teacher quality and quality teaching (e.g., Louden, Rohl et al., 2005a). A recent ACER study conducted for Teaching Australia (Ingvarson, Elliott et al., 2006) examined current procedures for the assessment and accreditation of teacher education courses. The findings indicated that these procedures are generally weak as quality assurance mechanisms. None is based on outcome measures of the quality of graduates or their competencies. There are over 200 teacher education courses in Australia, but, apart from one ACER study (Ingvarson, Beavis et al., 2005), we know little about the relative effectiveness of these courses. Clearly, there is a need to develop much better measures of the outcomes of teacher education courses if we are to understand the characteristics of courses that are more effective in producing competent teachers. ACER is currently coordinating an international study in 15 countries comparing the effectiveness of programs for preparing teachers of mathematics. This study includes the development of survey instruments that include measures of mathematical and pedagogical knowledge, which may enhance our capacity to measure the outcomes of teacher education course outcomes. (Further details can be found at: http://teds.educ.msu.edu/default.asp). Registration of new teachers is another important mechanism for ensuring teacher quality. Ideally, registration provides an assurance that new teachers are not only qualified but competent, but this is not the case in most states and territories. In most Australian States and Territories, registration follows automatically from completing an approved university qualification, despite the fact that this qualification alone is an uncertain guide to a teacher’s capacity to promote learning in real school contexts (Parliament of Victoria, Education and Training Committee, 2005). Most professions delay registration until a period of internship in workplace settings has been completed satisfactorily (Ingvarson, Elliott, et al., 2006). The Victorian Institute of Teaching has introduced new standards-based assessment procedures for provisional registration, which means that registration for teachers in Victorian schools now depends on successful completion of a period of provisional registration supported by a mentor. By the end of this period, graduate teachers are expected to provide evidence that their practice has met standards of performance established by the VIT before gaining full entry to the profession. These new procedures are perceived as valid assessments against the VIT standards (Ingvarson, Kleinhenz et al., 2007). Other states such as NSW are developing similar procedures. However, the success of these new procedures in promoting better teacher education and professional learning during induction will depend on the development of valid measures and standards of teacher performance. The foregoing indicates several reasons why it is important to improve our capacity to measure teacher quality in ways that are valid, reliable and fair. The focus of this paper is on 6 For specific details related to the PISA 2000 and 2003 results relevant to Australia, see: Lokan, Greenwood and Cresswell (2001); Rowe (2006b); Thomson, Cresswell and De Bortoli (2004). For TIMSS 2003, see: Martin, Mullis et al. (2004); Mullis, Martin et al. (2004); Rowe (2006b). Comparative findings from fitting explanatory multivariate and multilevel models to both the PISA and TIMSS student achievement data across Australia’s six States and two Territories have been reported by Rowe (2006b). _____________________________________________________________________________________ The Economics of Teacher Quality conference, ANU: 5 February 2007 Conceptualising & Evaluating 6 Ingvarson & Rowe Teacher Quality _____________________________________________________________________________________ recent developments in standards-based approaches to measuring teacher performance designed to address these purposes. In summary, these purposes include: • Accreditation of teacher education courses; • Registration of new teachers; and • Certification of accomplished and highly accomplished teachers. These purposes constitute the three key quality assurance mechanisms in any profession. They provide the answers to the following questions: ‘Who gets the right to train teachers?’ ‘Who gets to enter the profession?’ and, ‘Who gains recognition for attaining high standards of practice?’ If the rhetoric about improving and valuing teacher quality is to become a reality, these three fundamental quality assurance functions need to be operating effectively – functions that are best carried out at the national or profession-wide level. With some rare exceptions, there is little recent or current evidence to suggest that these mechanisms are operating effectively in Australia. This should be taken as a description of the current situation rather than a criticism of any particular group. This paper is based on the proposition that, to carry out these functions more effectively, we need to develop more rigorous methods of assessing teacher quality. Paradoxical though it may seem, more rigorous methods of summative assessment lead to better planning and formative assessment in teacher education and professional development (Ingvarson, 2003; Ingvarson & Kleinhenz, 2006a,b). If we are to develop methods for evaluating teacher quality for purposes such as outlined above, we need strong conceptual foundations for what we mean by teacher quality. The remainder of this paper focuses on methods for evaluating teacher quality for the purposes of developing a profession-wide system for identifying and recognising highly accomplished teachers. Conceptualising quality in teaching The guiding questions for this section of the paper are: ‘How do we develop valid indicators of teacher quality for purposes such as those above?’ and ‘How do we decide what teachers should know and be able to do?’ A closely related question is: ‘On what bases should teachers be evaluated?’ Another is: ‘For what is it fair to hold teachers accountable?’ These are questions that apply to all professions, and particularly with respect to medicine. On what foundations should teachers be evaluated? If measures of teacher quality are to be used in making decisions that are critical to teachers’ lives and careers, they should be based on valid criteria or defensible foundations. There is a long tradition of research on teacher evaluation issues in the USA. Millman and DarlingHammond (1990) provide one of the most comprehensive reviews of this research in their New Handbook of Research on Teacher Evaluation. Based on the work of Michael Scriven (e.g. Scriven, 1994), Wheeler (1994) provides a helpful classification of foundations or sources that have been used in the US for developing criteria for evaluating teachers, together with comments on their relative validity. These include: • Government regulations and requirements; • Professional standards; • Outcomes of teaching; • Theories grounded in practice; • What teachers are doing; • What others would like teachers to be doing; and • What teachers should be doing. The Appendix to this paper provides an elaboration of each of these sources. Each provides a way of answering the question: ‘How will we determine what teachers should know and be _____________________________________________________________________________________ The Economics of Teacher Quality conference, ANU: 5 February 2007 Conceptualising & Evaluating 7 Ingvarson & Rowe Teacher Quality _____________________________________________________________________________________ able to do?’ Each aims to provide a source for criteria to be used in determining the domains of performance and attributes that should be included in a system for evaluating teacher quality. Scriven (1994) and Wheeler (1994) weigh the arguments for and against, using each of these sources as a basis for evaluating teachers. They argue that, for employer purposes, such as performance management and decisions about retaining employment, the appropriate basis for evaluation of teachers is the last item, namely, what teachers should be doing, based on the duties and responsibilities of a teacher as should be delineated in an employment contract. However, for professional purposes beyond a single employer such as registration and advanced certification, a more appropriate basis for assessing teacher quality is what the profession says teachers should know and be able to do – as specified in a set of professional standards.